Skip to content

feat: Compare sample_entity_whole_population relative to "sentinel" CPU-bound benchmark#883

Merged
RobertJacobsonCDC merged 1 commit into
mainfrom
RobertJacobsonCDC_relative_benchmarks
May 20, 2026
Merged

feat: Compare sample_entity_whole_population relative to "sentinel" CPU-bound benchmark#883
RobertJacobsonCDC merged 1 commit into
mainfrom
RobertJacobsonCDC_relative_benchmarks

Conversation

@RobertJacobsonCDC
Copy link
Copy Markdown
Collaborator

@RobertJacobsonCDC RobertJacobsonCDC commented May 8, 2026

Motivated by #874, this PR adds a CPU-bound "sentinel" benchmark and compares the relative performance of sample_entity_whole_population in sample_entity_scaling. The idea is, even on different CPUs this relative performance will be stable.

Updated benchmark result processing to record relative_to_sentinel metadata for sample_entity_whole_population/{1000,10000,100000} when the sentinel is present. The dashboard now overlays that relative series on the same chart as absolute timing, with a right-side x sentinel axis, while hiding the sentinel as a standalone benchmark. PR benchmark comments also filter out the internal sentinel row.

Added test coverage for the result parser and PR comment formatter, including sentinel ratio recording and sentinel-row filtering.

@RobertJacobsonCDC RobertJacobsonCDC linked an issue May 8, 2026 that may be closed by this pull request
Investigate odd variance of whole population sampling #874
Closed
@RobertJacobsonCDC
Copy link
Copy Markdown
Collaborator Author

RobertJacobsonCDC commented May 8, 2026 •
edited
Loading

Because this PR technically adds a benchmark, the benchmark is expected to fail in CI.

Edit: We fixed this behavior in a recently merged PR.

@k88hudson-cfa k88hudson-cfa self-requested a review May 12, 2026 16:05
@RobertJacobsonCDC
feat: Added "sentinel" CPU-bound benchmark and compare `sample_entity…
a6a03d2 
…_whole_population` relative to this sentinel
@RobertJacobsonCDC RobertJacobsonCDC force-pushed the RobertJacobsonCDC_relative_benchmarks branch from 4ba7b51 to a6a03d2 Compare May 15, 2026 13:21
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 15, 2026

Benchmark Results

Hyperfine

Command Mean [ms] Min [ms] Max [ms] Relative
large_sir::baseline 2.8 ± 0.1 2.7 3.1 1.00
large_sir::entities 6.7 ± 0.1 6.4 7.2 2.34 ± 0.08

Criterion

Regressions (slower)
Group Bench Param Change CI Lower CI Upper
sample_entity sample_entity_whole_population 1000 58.871% 54.993% 61.997%
sample_entity sample_entity_whole_population 10000 54.954% 49.992% 59.436%
sample_entity sample_entity_whole_population 100000 34.388% 31.800% 37.091%
sample_entity sample_entity_single_property_unindexed 1000 16.087% 13.564% 18.559%
large_dataset bench_filter_indexed_entity 13.214% 3.085% 24.552%
sample_entity sample_entity_single_property_unindexed 10000 11.363% 10.199% 12.464%
large_dataset bench_filter_unindexed_entity 5.816% 2.230% 9.547%
counts single_property_indexed_entities 5.216% 4.176% 6.145%
counts multi_property_indexed_entities 4.492% 4.137% 5.025%
large_dataset bench_match_entity 3.377% 3.008% 3.760%
indexing query_people_multiple_individually_indexed_properties_entities 2.657% 1.551% 3.888%
Improvements (faster)
Group Bench Param Change CI Lower CI Upper
sample_entity sample_entity_single_property_indexed 100000 -8.281% -8.839% -7.745%
sample_entity sample_entity_multi_property_indexed 1000 -7.072% -7.507% -6.659%
sample_entity sample_entity_multi_property_indexed 10000 -6.609% -6.943% -6.178%
indexing query_people_count_indexed_multi-property_entities -6.600% -7.306% -5.938%
sample_entity sample_entity_multi_property_indexed 100000 -6.075% -6.484% -5.739%
indexing query_people_single_indexed_property_entities -6.063% -8.287% -3.776%
algorithm_benches algorithm_sampling_multiple_known_length -4.528% -5.146% -3.961%
sample_entity sample_entity_single_property_indexed 1000 -4.011% -4.666% -3.383%
large_dataset bench_query_population_derived_property_entities -3.664% -4.241% -3.174%
sampling sampling_multiple_unindexed_entities -3.354% -3.799% -2.910%
indexing with_query_results_single_indexed_property_entities -2.954% -3.517% -2.441%
sample_entity sample_entity_single_property_indexed 10000 -2.941% -3.611% -2.257%
sampling count_and_sampling_single_known_length_entities -2.550% -2.912% -2.076%
sampling sampling_single_known_length_entities -2.361% -2.602% -2.116%
indexing query_people_count_multiple_individually_indexed_properties_enti -2.139% -2.558% -1.763%
indexing query_people_count_single_indexed_property_entities -1.846% -2.025% -1.652%
sample_entity sample_entity_single_property_unindexed 100000 -1.614% -2.147% -1.116%
Unchanged / inconclusive (CI crosses 0%)
Group Bench Param Change CI Lower CI Upper
examples example-basic-infection 1.927% 0.746% 3.650%
examples example-births-deaths -1.795% -3.430% -0.516%
algorithm_benches algorithm_sampling_multiple_l_reservoir -1.188% -1.399% -0.995%
large_dataset bench_query_population_multi_indexed_entities 1.163% 0.744% 1.511%
large_dataset bench_query_population_multi_unindexed_entities -1.095% -1.740% -0.334%
counts index_after_adding_entities -1.090% -1.207% -0.978%
sampling sampling_multiple_l_reservoir_entities -1.071% -1.521% -0.417%
sampling sampling_multiple_known_length_entities 0.747% 0.354% 1.265%
sampling sampling_single_unindexed_concrete_plus_derived_entities -0.629% -0.835% -0.441%
algorithm_benches algorithm_sampling_single_rand_reservoir -0.500% -0.884% -0.176%
indexing with_query_results_multiple_individually_indexed_properties_enti -0.484% -1.043% -0.005%
indexing with_query_results_indexed_multi-property_entities -0.455% -1.085% 0.080%
sampling sampling_single_unindexed_entities 0.406% 0.341% 0.484%
algorithm_benches algorithm_sampling_single_known_length -0.254% -0.728% 0.219%
large_dataset bench_query_population_indexed_property_entities 0.237% -0.082% 0.678%
large_dataset bench_query_population_property_entities -0.230% -0.570% 0.074%
counts reindex_after_adding_more_entities 0.179% -0.110% 0.506%
sampling sampling_single_l_reservoir_entities -0.166% -0.785% 0.281%
counts multi_property_unindexed_entities -0.163% -0.437% 0.094%
indexing query_people_indexed_multi-property_entities -0.142% -0.569% 0.284%
counts single_property_unindexed_entities -0.086% -0.627% 0.319%
sampling count_and_sampling_single_unindexed_concrete_plus_derived_entiti -0.084% -0.249% 0.112%
counts concrete_plus_derived_unindexed_entities 0.028% -0.675% 0.695%
algorithm_benches algorithm_sampling_single_l_reservoir -0.014% -0.165% 0.105%
Not Compared (no baseline yet)
Group Bench Reason
sample_entity_whole_population sentinel No baseline named 'base'

github-actions Bot added a commit that referenced this pull request May 15, 2026
@github-actions
Update bench-history.json (PR #883 @ a6a03d2)
7061608
@RobertJacobsonCDC RobertJacobsonCDC merged commit a62301a into main May 20, 2026
22 checks passed
@RobertJacobsonCDC RobertJacobsonCDC deleted the RobertJacobsonCDC_relative_benchmarks branch May 20, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@k88hudson-cfa k88hudson-cfa k88hudson-cfa approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Investigate odd variance of whole population sampling

3 participants

@RobertJacobsonCDC @k88hudson-cfa @uhc7